Express Mail #EE633279672US 
Docket Number: 20429/5 



Application for U.S. Patent: 



TEMPLATE CREATION AND EDITING FOR A MESSAGE CAMPAIGN 



Inventors: 
Seth Haberman 

and 
Chet Schuler 



Correspondence Address: 

David D. Lowry (Reg. 38,538) 
Brown Rudnick Freed & Gesmer 
Box IP, 18* Floor 
One Financial Center 
Boston, MA 02111 
Phone 617 856-8399 
Fax 617 856-8201 




Client/Docket Number: 20429/5 
TEMPLATE CREATION AND EDITING FOR A MESSAGE CAMPAIGN 

5 

FIELD OF THE INVENTION 
This invention is directed towards video and media creation, and more particularly 
towards a system for creating personalized messages based on user information. 

10 BACKGROUND OF THE INVENTION 

Marketers have come to rely on demographic solutions to establish pattems and trends 
about the purchasing habits of their customers and how these habits relate to media purchases, 
^3 demographics, and other factors. Alongside companys* proprietary databases, third party data 

i f\ 

warehouses have evolved, fashioned by many companies who share information either about 

i r3 

i'k* 15 specific customers or about data extracted from their customer bases. In both cases, advertisers 
^ y use the derived information to generate observations relating to their markets, target individuals 

ii to different types of offerings and select appropriate media purchases for advertising. 

■•^ 

3. In the case of video advertising media, e.g., video tapes that are mailed, internet video streams, or 
broadcast or cable advertising - there are only limited means to produce personalized versions of 

^jf 20 the advertisements or direct marketing information that directly takes advantage of the 

information available about consumers purchasing habits and the like. Mostly, this reflects the 
nature of the traditional delivery media for video, which, until very recently, provided very 
limited capability to deliver anything more than a common message. As such, even though 
companies know a tremendous amount of information about their customers, the ability to 
25 leverage this information has been limited by the fundamentals of the broadcast medium, 
requiring a like message to be sent to all customers. Additionally, the nature of video 
production, focused typically on one product, does not allow for incremental content changes. 
This forces advertisers to avoid topically relevant information and offerings. 

An early attempt to merge the power of real-time television with the information content 
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of the Internet is disclosed in U.S. Pat. No. 5,778,181, Hidary, et. al, dated July 7, 1998 (the 481 
patent), in which a user is presented with a video along with related Internet information. A user 
has a personal computer connected to two signal paths: a video signal and a second separate 
connection to the Internet for receiving Intemet information. To use the described system, the 
5 video producer needs to first create the video, encode the Intemet addresses (or URLs) of web 
pages, and then distribute the composite information. A limited portion of the vertical blanking 
interval (VBI) of the video signal is encoded to deliver the URLs. The client software retrieves 
and decodes URLs from the video program. These URLs are interpreted, and direct the web 
browser software to retrieve web pages. The web pages are synchronized to the video content for 
10 display. Alternatively, the encoded video signal could be simuhaneously routed to a specialized 
server that decodes the URLs, contacts the addressed Intemet server, and directs the server to 
send the designated web pages via the Intemet to the user's PC where they are then displayed. 
^.0 Among the disadvantages of the system such as described, is that the video and corresponding 

In 

;;p content are static and defined by the video producer during production. The same URLs are 

j'L! 15 sent to every viewer of the same video. There is no change to the content of the video according 
to any real-time understanding of the viewer. Similarly, no mechanism is described for changing 

ttpa 

i: the content of the web pages in real time either, even if the server somehow obtained information 

3 about the preferences of the viewer. 

''"^ A somewhat selective system is described in Abecassis, U.S. Pat. No. 5,717,814, issued 

H 

Q 20 Feb. 10, 1998. The system in the *814 patent includes a capabiUty for a more individualized 

•'3 

control over the contents of a requested video program. Segments, or clips, of a video scene are 
first evaluated and rated according to criteria such as the amount of violence, nudity, or 
profanity, as well as other criteria related to other elements of the clip (location, time, detail, 
etc.). Each clip is associated with a record containing keywords related to the subject matter, 
25 such as "flag buming," that a user may prefer not to see. A series of clips (a "program") is then 
mapped into a series that constitutes the presentation when viewed in sequence. During run- 
time, the actual series of clips shovra are selected from the library of clips according to the user's 
preferences. Thus, a viewer might decide to watch a condensed version of a movie without any 
gore while another viewer may prefer a lengthier version with gore but not profanity. A producer 
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and director can also create multiple versions of important scenes that will be viewed according 
to the viewer's preferences. As the price of such user-selection, this system not only requires the 
producers initially to rate, key, and map each segment in a movie according to the various 
parameters, but also requires a specialized viewing system that can translate the viewer's 
5 preferences into the proper selection and mapping criteria for properly sequencing the optional 
segments during the playback. For example, a specialized laser-disk unit would be programmed 
to play a specially encoded disk according to the viewer's desires. In a wideband network 
context, the transmission system would first upload user preferences, omit non-conforming clips, 
and create the resulting download movie. In any case, the result is a pre-selected set of clips, as a 
10 fimction of the user preferences, that will be played in sequence for the viewer. The creation and 
inclusion of clips in the program must necessarily precede consideration of the user's 
preferences. The viewer is not really interacting with the producers, but only the machine 
=:Q making the selection of pre-selected clips. No "new" materials are selected or incorporated into 
V the show during viewing - only "old" materials that have been pulled together as a function of 
i'J=j 15 the user's selected preference. Thus, nothing is "created" specifically for the viewer. 
^ ^ A somewhat more interactive solution is described in Freeman, U.S. Pat. No. 5,861,881, 

I: issued Jan. 19, 1999, in which a user interacts with a computer to determine which of several 

p cable TV or other audio-visual inputs will be displayed on the screen at any given time. Stored 
user commands determine which of the multiplexed signals will be selected, and specialized 

'"•4 

□ 20 hardware "seamlessly" switches between video channels, such that the user cannot perceive the 

C3 

switchover (other than the change of content). For example, at the start of a sports program, the 
user can select the preferred language of the audio segments, and whether on-screen graphics 
(e.g., player statistics) should be displayed. These inputs can be used to insert scheduled "trigger 
points" into the video or digital program which are later used to select the audio-visual 
25 combinations preferred according to inputs by the user. When a trigger point is played, the 
computer system selects a different video segment, graphic, or audio segment, either fi-om 
alternate channels, or from a database of such segments downloaded to the computer on a second 
communication channel (either in parallel with the primary transmission, or on CD-ROM, or at 
some other time). As with other versions of this approach, the user must intentionally interact 
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with the program to select the preferences. Also, the only selectable segments are those 
produced as part of the program. There is no real-time search for or compilation of new 
materials. The user preferences are only evaluated according to the pre-selected criteria that 
correspond to segment selections. Each video segment is either played or not played; but the 
5 video segment itself is not modified according to any user inputs. 

The television industry's reaction to the limitations of broadcast medium has been 
"localism," that is, an attempt to target audiences by demographic, geographic and psychographic 
means. Localism reflects market generalizations, often derived fi-om actual profiles and data. 
Localism is offered in the broadcast forum by buying placements in geographic areas. On cable 
10 this can be a neighborhood. On specific television shows it can be via reflecting certain viewer 
preferences according to demographics, or times of day. The results have left advertisers, who 
pay by the "eyeball," dissatisfied with the expediency of their advertising dollars. The cable 
'tO television industry, in an attempt to respond to these shortcomings and to distinguish its offering 
fi-om broadcast, is deploying addressable advertising systems which allow advertisers to purchase 
15 individual households or even single set-top boxes for their ads, allowing, when combined with 
^ y household profiles, an extremely fine grain of targeting to reach the appropriate viewers. 

iib;: 

•:i This form of targeted or addressable advertising, is similar to the capabilities now found 

^ p on some Internet services, where single viewers based on their profiles can be targeted with 
;H banner ads and interstitials or email. At the same time, Internet advertising is increasing its use 
□ 20 of rich media (defined as 20-Kbytes/sec or above), in an attempt to create the type of powerfiil 
and effective images we associated with television advertising. Sometimes because of 
bandwidth limitations and often because of a mistaken philosophy, these ads necessitate 
consumer interactions. The result, in contrast to television advertising, forces the consumer to 
interact, distorting the media's capability to dehver a message. 
25 As a means to improve on targeting, designers of Web based offerings began to employ 

personalization solutions. On the Intemet, where many traditional factors associated with 
purchasing decisions become less relevant, the ability to establish a one to one relationship with 
a consumer is paramount. Similarly, it is most desirable to offer, if possible, service and 
products reflecting that one-on-one relationship. To cite an example, Dell Computer allowed 
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customers to build their own computers (mass customization) on the Web, instead of pre- 
building several different models for different target markets (targeting). Dell and other 
companies employed sophisticated customer relationship management (CRM) solutions based on 
general and proprietary technologies. 
5 These CRM solutions, based on a variety of existing technologies, personalize offerings 

by using a combination of information gathered from the consumers visiting web sites. Such 
information is typically gathered by asking questions, tracking navigation and purchasing 
behavior, as well as from information gathered elsewhere. 

Yet another step in the enhancement of advertising has been taken recently, as reported in 
10 the press. For example, Enliven of Waltham, MA, offers real-time, database connection 

capabilities that let advertisers present up-to-the-minute information in Enliven-activated ads. 
Advertisers can present live information from a database source to a consumer viewing the ad 
with a proprietary, Java-based viewer. When merged with a marketing database such as 
,p TrueMatch, profile-based ad campaign targeting becomes possible. An advertiser creates a 

) s I 

i'j=i 15 template into which graphics and text are inserted, according to the demographic information 
^ y available about the user. However, even with the additional consideration of demographic 

information for real-time selection of advertising components, the templates provide a limited 
capability to effect enhanced advertising. This known system does not include any capability to 
1^1 modify or incorporate video materials into the advertisements, thus depriving the advertisers of 
^=3 20 the rich menu of video-oriented raw materials from which to draw. This leaves the viewer with 
little more than an interactive baimer ad, perhaps with primitive "effects" that happens to have 
been "tailored" for publication to that user. There is no capability to change the tempo, music, 
narration, lighting, or any other elements of the advertising that have been proven successful in 
gaining the viewer attention the advertisers desire. 
25 Further, even with known interactive personalization solutions in place, text and graphics 

solutions cannot compete with the masterful ability of video and audio to generate interest, create 
brand awareness, or product image. Television advertising agencies are master storytellers, using 
the types of narrative that people respond to so well. Although the Internet is a powerful medium 
on which to personalize dynamic offerings it lacks the power of traditional television which uses 
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the richest of media to present powerful 30-second stories. The Internet is, by form, an 
interactive media, with the user typically controlling the type of message received. If the user 
can control the media message, either through explicit commands about what type of information 
the user reveals, or alternatively what information the user receives, the power of narrative story 
5 telling to a passive and receptive audience is lost. 

SUMMARY OF THE INVENTION 
The present invention is directed towards a system and method for dynamically creating 
individualized, multi-media messages for delivery to an intended audience, which can be specific 
10 groups or individuals. A message, story, or advertisement is assembled on demand, based upon 
rules applied to each user's profile data and the available library of media segments. The 
narrative framework for the final personalized message is a story as defined by a message 

^0 campaign. The message campaign includes a message template and a collection of media 

1 n 

segments. The media segments are selected and then assembled to produce the final personalized 
m 15 message at assembly time. Specific media segments are selected and merged according to the 
' y message template and information about the viewer derived fi-om a user profile. The information 

! 

fi-om the user profile is interpreted by an expert rule system to determine which of several 
:,E potential media segments to select for use in the personalized message. The merged composite is 
■rl then encoded to match the distribution media and forwarded to the user for viewing. 
Q 20 Although the operation of the present invention includes the ability to assemble and 

convey a message upon demand, an advantage is the ability to match the presentation options 
with an educated and timely assessment of the target audience according to a number of 
templates. A campaign plan defines what the target group is (entity qualification), and what 
individual viewer information is relevant (entity profile) for target entities within the target 
25 group. Additional factors include the selection of input databases, target distribution channels, 
and environmental factors (weather, current interest rates, etc.). A message resource library 
contains the collection of video, audio, and other elements necessary to assemble the whole range 
of different messages based upon the message template. Message library resources include not 
only the varied clips necessary for each selection point of the template, but also variations 
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triggered by changes in the monitored environmental factors, as well as synthesized speech and 
visual constructs to meet individual profile requirements. 

Assembly of the message according to the present invention is based upon a message 
template which embodies the rules for selection and combination of the most current message 
5 library resources according to the entity profile at the scheduled time of message production. 
The present invention allows automated dynamic message assembly at any point up to seconds 
before delivery, based upon entity and environmental factors that are in constant flux, yet 
combined in a manner that addresses the communicative objectives of the campaign. A master 
task scheduler is defined according to the campaign requirements to control and coordinate all 
10 activities of the creation of the viewer profiles, message resource library, and message assembly. 

The present invention includes the ability to edit and use scene or segment substitutions, 
choice of narrators and language, music changes or substitutions, audio effects, text and graphic 
v3 overlays, speed changes, background changes, word or phrase insertions, more complex story 

i - I 

changes, utilization of video libraries, digital video effects (DVE's) and transitions, and seamless 

ih 

i'^^ 15 synthetic voices, characters, and studio sets. Many of these scenes or segments may also be 
J modified at message assembly time according to the selected environmental inputs, or 

=j interpretation of individual viewer inputs. 

Q 

^ r Advantages of the present invention include message creation upon order, not by forecast, 

;;1 with story-driven message assembly tailored to the individual and any combination of 
□ 20 information known about the target viewer and the present environment. A new marketing 

paradigm is created, with the focus on differentiating customers, not just products. This in turn 
creates new opportunities for increasing customer interest, satisfaction, and retention of the 
campaign message. 

Other advantages include the automated real-time creation of the message, which solves 
25 problems of timeliness and personal privacy. Computers, not humans compose the final form of 
the ad fi-om the raw materials and previously generated templates. 

Other advantages of the present invention include a system and method for gathering user 
profile information fi-om a variety of different sources and databases, and processing the user 
profile information by a profiler which uses rules of assembly for creating a format usable by the 
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message creation system. 

Other advantages of the present invention include a system for creating templates with 
multiple insertable media features to create a personalized message for an individual or group. 
The number or type of insertable media features is not limited. For example, every element of a 
5 television commercial may be selected (or dynamically created) to fill in a template, including 
background view (such as a city skyline, and seasonal choices as well), music (background and 
jingles), language and accent of narrative, the choice of what narrative to add, the product being 
shown (for example a car or truck), the appearance of the product (for example the color of a car 
or accessorizations), selectable video of real actors, the length the conunercial runs, any screen 
10 over text, etc. The media segments may all be created with the initial template, or created 
afterwards, including new media segments based on current events, but which are used in a 
template which has been in use for some time. 

Another advantage is an expert rule base which allows the selection of proper media 

in 

'^Q features for an individual, based on collected information about the individual. The expert rule 

i 

15 base is able to use incomplete data or knowledge to make appropriate decisions about selecting 
^ H media segments for an individual. The expert rule base can also make inferences about an 
* individual based on available data. 

3 Yet another advantage of the present invention is a highly flexible delivery system and 

::i method. The media messages may be assembled at any time during the process, anywhere fi-om 
Q 20 the time the message template and media segments are created, or up to and including real-time 
delivery where the media message is created and shown to the individual. Further the media 
message may be created at many different places, both centralized and decentralized, fi-om the 
studio, to local station or web site, on a satellite, at a syndication station, at a cable television 
central office, neighborhood network, or even by a satellite receiver or cable box inside a 
25 viewer's home. Still fiirther, the choice of delivery mediums is extremely varied, fi-om 

prerecorded video tapes, DVDs, CDs etc. which are sent to an individual, to live feed through a 
cable system, internet connections, satellite link, RF towers, line RF signals, cell phones etc. 

BRIEF DESCRIPTION OF THE DRAWINGS 
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The foregoing and other features and advantages of the present invention will be more 
fiilly understood from the following detailed description of illustrative embodiments, taken in 
conjunction with the accompanying drawings in which: 

Fig. 1 is a block diagram of a system according to the present invention; 
5 Fig. 2 is a flow chart showing an overview of timing of media creation according to the 

present invention; 

Fig. 3 is an overview of assembly of components to produce a media message according 
to one embodiment of the present invention; 

Fig. 4 is a more detailed diagram of assembly of components as overviewed in Fig. 3; 
10 Fig. 5 is a block flow diagram of an illustrative embodiment of a system according to the 

present invention; 

Fig. 6 provides details of an example message template according to the illustrative 
embodiment of Fig. 5; 

; cz: 
1 t : 
: 

£ Fig. 7 provides details of an example message resource library associated with the 

i h 

in 15 message template of Fig. 6; 

' y Fig. 8 illustrates assembly of a message according to example rules applied to the 

components of Figs. 6 and 7; 

Fig. 9 is a block diagram of an embodiment of the present invention for personalized 

.tin. 

'Ci message delivery over an intemet system; and 

0 20 Fig. 10 is a block diagram providing more details of the embodiment of Fig. 9. 

1 : 
'iBtr 

DETAILED DESCRIPTION 
A system 20 for personal message creation and delivery according to the present 
invention is shown in Fig. 1 . A user profile database 22 includes information regarding an 
25 individual 24. This user profile database 22 can be in any form, including a proprietary database 
of information owned by one entity, or publicly available information at one or at multiple 
locations, including information from user interactions on web sites or shopping networks. For 
example, the database may be a subscriber database accessed by the system 20 on a per- 
transaction system. The user profile database may contain any type of information regarding the 
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individual 24, including demographics, address, monetary income, political affiliations, known 
preferences, buying patterns etc. 

A template database 35 includes message templates which reflects the story, or message 
that the message creator or provider desires to convey to the audience. This message is described 
5 in terms of a "campaign", in that the message may be any type of message from a product 
advertisement, to a political message or informative message which may be tailored for each 
individual or group receiver. Details regarding the creation of a campaign will be described 
below. 

One or more libraries or databases 26 include media segments which are used to assemble 
10 the personahzed message. The databases 26 include a compendium of elements that may be 
broadly categorized as graphics 28, video and sound segments 30, and animation 32. These 
,™ media segments may be part of a general library of available material, for example pictures of 
:=2 individual city skylines, attractions, or natural scenery for use in backgrounds. Alternatively, 

j : I 
■[? : 

£ media segments may be generally or specifically created for a certain message campaign. For 
Iji 15 example, in an automobile advertisement, several media segments showing a certain automobile 
* J model in different colors may be provided, which are to be inserted into a media template from 

the template database 35 at an appropriate location. Which media segment is selected for 
up insertion depends on user 24 information, together with the media template specifications, as will 
^3 be discussed below. 

i;3 20 Similarly, the database for video and sound 30 and animation 32 may include both "stock 

footage" which are available as needed by any message campaign; and specific message 
campaign material created and stored for a particular campaign. Animation includes any type of 
animation such as cartoon characters and logos etc. 

In addition to the resource library, the system according to the present invention is able to 
25 synthesize additional elements 34 as needed during message creation, thereby providing 
increased flexibility. Such synthesized elements include sound such as synthesized speech, 
music, background sounds, and graphics such as text, background filler, visual objects (including 
color variations thereof), and visual effects (including dissolves, morphing of objects, etc.). 

The personahzed messages are assembled by an assembly system 36. The assembly 
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system receives a message template from the template database 35, and uses media segments 
from the databases 26 to put together the message. The assembly system 36 receives input on the 
user 24 from the user profile database 22, and also receives input from expert rules 38, which 
interpret the user profile data, and direct the assembly system 36 to select which particular 
5 segments from the databases 26 to combine for the personalized media message. The expert 
rules 38 system is capable of interpreting user profile data from many different sources and in 
many different formats. The expert rules 38 system is also capable of making decisions based on 
incomplete user profile data, providing logical "fallback" selections as necessary. The expert 
rules 38 system will be described in detail below. 
10 The assembled personalized message is encoded for a delivery medium 40, and then 

dehvered to the individual viewer 24, typically by a display device 42, which can be any of 
various types of receiver including television, computer monitor, radio, phone etc. The delivery 
I^B medium can be any of various types of media, including prerecorded video tapes, DVDs, CDs 
etc. which are sent to an individual, or live feed through a cable system, internet connections, 
I n 15 satellite link, RF towers, line RF signals, cell phones and the like. As the present invention 
* P facilitates real-time creation, personalized messages to individuals may be delivered in many 
different ways, based on the user profile. For example, an individual 24 with a sateUite 

s 

iter 

:£ connection would receive a distribution in a format different from another individual 24 
'Jj receiving a media message for the same campaign, but delivered by the internet. More detail 
J 20 regarding the delivery medium 40 will be provided below. 

A campaign message process according to the present invention are outlined in Fig. 2. 
Campaign message programs are divided in three basic stages: message creation time 50, 
assembly time 58, and delivery time 64. During the campaign creation time 50, the producers 
define and create the message templates 56 and also create or select media segments 54 the 
25 system will need for generation of the personalized messages or advertisements, step 52. The 
message template 56 is the framework which holds the story together. Results of the creation 
step are stored in the template database 35 and the message resource library 26 as a set of 
message templates 56 and media segments 54. 

At message assembly time 58, the present invention uses information from the user 
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profile 60, message template 56 and media segments 54 to create the personalized message for 
the specific user 24, step 62. The system uses expert rules 38 to assist in the process. 

The next step is message delivery time 64, where the personalized message is presented 
to the user 24. This presentation may involve transmitting the message, or placing the message 
5 on some type of media. As previously described, message creation time 58 and message delivery 
time 64 may be contemporaneous, in that the message is created on the fly, or "just in time". 

Fig. 3 illustrates an embodiment of the invention in which substitution editing is 
implemented. A producer or editor working at an editing station 70 creates a message template 
56 comprised of descriptions of different types of multi-media materials such as video, graphics, 
10 music and narration. The editor also creates or selects the media segments and consoHdates 
them in the selected storage or database 54. User profiles 60 about the user 24 are obtained and 
fed to a rules system 38 for substitution editing. The rules system then selects raw materials 
from the media segments database 54 according to the message template 56 and user profile 60 
and combines them to form a completed personalized message 72 that is deUvered to the viewer 



m 15 24. 

^ p Fig. 4 provides a more detailed view of the assembly stage according to an embodiment 

of the present invention for rules based substitution editing. A message template 56 is obtained 

i 

fi-om the message template database 35. The message template 56 describes a fi-amework to 
create and complete a personalized message 72 for the selected individual 24. The message 
•;3 20 template 56 runs for a certain length of time, as shown by arrow 76. The final running time of 
the personalized message may vary, as according to the present invention, the message can be 
diversified on many levels, including short messages or very long messages, as appropriate for 
the individual 24. 

The message template 56 may include both predefined sequences 78 and insertable 
25 sequences 80. Predefined sequences 78 are media segments which are conmion to all final 
personalized messages 72, in that they do not vary based on the individual 24. In the example 
message template 56, the first sequence 78 is a predefined sequence. In this presented 
embodiment, the predefined sequences are stored in the media segment database 26, and inserted 
at the appropriate locations in the message template 56. The predefined sequences 78 only differ 
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in the same predefined sequence 78 is always selected for that location in the personalized 
message. Alternatively, the predefined sequences may be stored or linked directly into the 
appropriate location in the message template 56. 

The other sequences are insertable sequences 80, which are filled in as directed by the 
5 rule system 38. The rule system 38 receives information fi-om the template database 35 

regarding the message template 56 being assembled, and also receives user profile information 
22. The rule system then determines the appropriate media segments to insert into the insertable 
sequences 80 of the message template 56. For example, the first insertable sequence 80a is video 
selected fi"om the video and sound database 30. A next insertable sequence 80b is also selected 
10 fi-om the video and sound database 30. Continuing with the present example, a predefined 
sequence 78 is then followed by an insertable sequence 80c comprising animation from 
...^^ animation database 32. The next insertable sequence 80d is another video segment from the 

■itsf 

video and sound database 30, followed by a final insertable sequence from the animation 

\j\ 

:S database 32. 

I'H 15 Also the predefined sequences 78 and insertable sequences 80 are shown in the present 

' t example with no overlap, however the present invention is not limited to orthogonal assembling 

JICS3 

•[^ of media segments. All elements of the personalized message can be controlled, and combined 
.:P in various forms to provide powerfiil customization. For example, a background scene 86 such 
!j as a geographic landscape, is selected from the video and sound database 30 and used as a 
20 common background for the predefined segments 78 while insertable segments 80 are added to 
the message template 56. The background scene 86 is not used for the end of the personalized 
message, in that the insertable sequences 80d and 80e do not use a background (or aheady have a 
background). In similar techniques, various physical elements may be inserted into standard 
video, for example the video and sound database 30 can include footage of several different 
25 actors providing dialog for a message, and the rule system 38 selects footage of a certain actor 
based on information from the user profile 22. The selected actor footage is then added as 
foreground in an insertable sequence 80. In this way, every element of a personalized message 
can be customized. 

Additional elements such as music 84 can be added to the personalized message as 
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appropriate. In the present example, music 84 is supplied for the end of the personalized 
message. The rule system 38 selects among the choices of music provided for insertion into the 
message template 56, and music is added from the video and sound database 30. 

Another element provided for the message template 56 is narration 82, which for the 
5 present example is synthesized voice 34 which is created dynamically as needed. Alternatively, 
prerecorded speech can be used, including multiple selections of prerecorded speech, which is 
selected according to criteria processed by the rule system 38. For example, prerecorded speech 
may be in different languages, or with regional accents, allowing the present invention to 
customize the message with different rich media features for the message creation. 
10 Fig. 5 shows an illustrative embodiment in which additional features of the invention are 

described, with details of the process of creating a campaign. Prior to the creation of a message 
or advertisement, a client works with the system operators to determine a formalized campaign 
plan 100 to be executed. The campaign plan is used by the client to define the communications 
=..C task the chent wishes to accomplish and, in turn, it drives all of the processes necessary to 

in 

in 15 achieve the program. 

I J J 

' ~ The first step is to create rich media video composition example or examples of the 

■l^ message that the client wishes to deliver to the audience. Typically it will be of the prescribed 
::p time duration and exhibit one complete message as an example of one specific version of the 
intended commimication. This provides the starting point for later construction of a message 

.ICS 

20 template or templates 56, Fig. 2 and a resource library (media segments 54) from which the 

y 

various personalized versions of this message are assembled. 

The next step involves target entity qualification data factors 104 Fig. 5. The client 
delineates the general characteristics of the members of the audience to which the message is to 
be delivered. It may include, for instance, appropriate media connection, family status, financial 
25 category, age grouping, regional location, etc. This information is used in the database searches 
to acquire the list of individuals or entities to which the message will be distributed. 

The next step involves search databases selection 106, and allows the client to define the 
limits of the search by identifying the databases 200 which are to be searched for data input. The 
entity profile database itself can be an input source built up interactively with the target entity. 
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The next step involves creating an entity profile template 108. The entity profile template 
108 is a complete or partial definition of the information about each individual or entity that is to 
be acquired by the search as well as a pattern for the formatting of this data. It is used in the 
generation of the target entities profiles and status database which not only govems the resources 
5 and rules used to assemble each personalized message, but also is the repository for all of the 
interactive information received fi'om the target user 24. 

The next step, called distribution channels selection 1 10, is defined by the campaign. It 
allows the client to decide what media outlet or outlet priority he wishes to use to communicate 
with his target audience. 
10 The next step is the definition of the delivery window specifications 112. These 

specifications define a time duration over which the campaign will be executed as well as if 
special time and/or programmatic related conditions are required. This information is the starting 
point for generating a master task schedule 118. 

Ul 

:S The next section involves interactive query response 114, whish is used to define any 

I n 

m 15 interactive transactions that may be utilized by the campaign such as acquiring additional 
' P information about the target entity, polling responses, purchase transactions, etc. 

The next section involves environment status factors 116 and refers to establishing a 
:.p description for existing environmental factors, for example weather, current interest rates^ current 
y travel fares, etc. that need to be updated at the time the message is transmitted to enhance the 
20 timeliness and relevance of the personalized communication packets. 

The final section involves verification transaction financial factors 118, and is used to 
define all of the financial factors such as verification, billing, purchase transactions, etc. which 
must be monitored 201 and recorded 502. Also any actions to be taken as a result of this 
information are indicated in this section 118. At this point, the client's work in configuring a 
25 campaign plan 100 is completed. 

The next set of actions are directed towards preliminary service activities. The first step 
204 in this process is to assemble the target entities profiles and status database 503. This is the 
working database for this illustrative embodiment. It relies on the database selector 200 which is 
driven by the campaign plan 100 to determine which databases 200 to search. This database 503 
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should be completed before any of the other activities may proceed since the information it 
contains is used by the other activities to complete their tasks. It may, however, be updated at a 
later time by the ongoing service activities including the query response tasks. The search is set 
up utilizing the target entity qualification factors 104 to decide which data records to retrieve and 
5 the entity profile template 108 to select and format the data for each individual or entity. It 
reserves fields for whatever polling, verification, transaction, or financial etc. information is 
required. This database 503 is the repository for all of the information generated by this 
embodiment about the target entities. 

The next step 208 to be defined is the management of the query and response interface 
10 with the target entity. When present, this activity allows two way conmiunication on a limited 
basis with the individual 24 who receives the message. It typically requires special capability 
r-^. embedded in the media distribution network. It also needs to be able to distinguish the identity 
;-| of the sender and associate his responses with his data profile. Timing is coordinated by the 

Master Task Schedule 308. 
m 1 5 The process of determining the message template 400 is an important part of this 

| p embodiment. The template embodies the rules for assembling personalized messages utilizing 
the profiles data and the resource library 26 Fig. 1 . An operations diagram for this activity will 

::p be described in reference to Fig. 6. It typically relies on the range and detail of the formatted 

'•'11 

y profile information, the campaign plan example message, and the requirements for any 

20 query/response activity that is specified. 

; 5 

•tt=T 

Once the message template has been constructed, then a message resource library 26 is 
created, step 300 Fig. 5. This resource library 26 contains the alternate clips of video and audio 
needed to assemble the whole range of different messages. One sub-process 304 has the 
instructions to receive environmental information, and synthesize the video and/or audio 
25 segments required by the message template for timely individualized message assembly. Another 
sub-process 302 provides the ability to synthesize, on the fly, artificial speech and visual 
constructs to meet the individual specifications indicated by the different entity profiles. 
Obtaining and maintaining current environmental information is a separate parallel operation 306 
that is constantly polling the appropriate information channels and making the acquired data 
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available on demand to the unit 304 that synthesizes this status information for the resource 
library. This information can be obtained from places such as the internet, the weather service, a 
news service, a private intemet or intranet, etc. 

A next step 308 is to establish the master task schedule. The information gathered by the 
5 campaign plan 100 is formatted such that it outlines the process flow and coordination 

requirements for each step of the ongoing service project from initiation to completion. This data 
is utilized to establish the master task schedule which controls and coordinates all of the 
activities and events as soon as its project "clock" is started. 

The next set of actions are directed towards service execution and delivery. After the 
10 preliminary service activities have been executed (except for timely updates) the schedule 
"clock" may be started. This controls the scheduled generation and delivery of the projects 
personalized messages to all of the profile entities in the target entities profiles & status data base 
;i3 503. Execution is a continuous step and repeat process which can deliver on the fly or 
accumulate appropriate batches for simultaneous delivery. Each step begins with the 
\n 15 presentation of the next profile 206 to the assembly node 404 which uses this profile to select and 
' ^ assemble the correct material from the resource library into its "slot" in the appropriate selected 
=j message template. It then sends the assembled message on to the transcoding and/or compressing 

,r operation 406 which delivers a video stream that matches the specifications for the delivery 
i;1 media. These video messages are transported to the chosen media distribution center 212 for 
^ 20 delivery at the scheduled time to the media user interface 214 which communicates with each 
appropriate target entity 214. 

When creating the campaign plan, the campaign manager working with the client to 
encode a default message example 102. The example serves as a message template containing a 
rich media video composition of the message that the client wishes to deliver to his audience. 
25 The example is of the prescribed duration and exhibits one complete advertising or other 

message as an example of one specific version of the intended communicafion to be dehvered. 
This provides the starting point for later construction of a message template 400 and a resource 
library 300 from which the various personalized versions of the message are to be assembled. 

The client delineates the general characteristics of the members of the audience to which 
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he wishes his communication to be delivered. It may include for example appropriate media 
connection criteria, family status, financial category, age group, regional location, etc. The 
resulting set of target entity qualification data factors 104 is used in database searches to acquire 
the list of individuals or entities to which the message will be distributed. Further criteria for 
5 search database selection 106 are also defined by the client. 

An entity profile template 108 is also created as a complete definition of the information 
about each individual or entity that is to be acquired by the search as well as a pattem for the 
formatting of the retrieved data. This entity profile template 108 is also used in the generation of 
the target entities profiles and status database 204 which not only governs the resources and rules 
10 used to assemble each personalized message but also is the repository for all of the interactive 
information received fi^om the target entity. A distribution channel selection 110 defines itself 
,^ This selection allows the client to decide what media outlet or outlets he wishes to use to 
B communicate with his target audience. 

p Client campaign plans also define the delivery window specifications 112. These 

fi 15 specifications define a time duration over which the campaign will be executed as well as when 

special time or programmatic related conditions are required. This information is the starting 
1^ point for generating a Master schedule 308. 

,E Interactive query responses 1 14 are defined where the campaign desires to acquire 

; J additional information about the target entity, poling responses, purchase transactions, etc. The 
3 20 client can also define environmental status factors 1 16. These factors will need to be updated at 
the time the message is transmitted to enhance the timeliness and relevance of these personalized 
communications packets. Examples include the weather in various places, current interest rates, 
current travel fares fi"om place to place, etc. 

As a final preparation for operations, the client and system operators define all of the 
25 verification transaction financial factors 118 which must be monitored and recorded. This 
element also includes definitions for any actions to be taken as a result of this information. 

Once the campaign plan has been encoded, a set of preliminary service activities takes 
place in preparation of the elements necessary for the dynamic customization phase. The first 
activity is to assemble target entities profiles and status database 204. This is the working 
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database for the whole current dynamic customization system. It rehes on the database selector 
200 which is driven by the search database selection 106 to determine which databases to search. 
This database 204 must be completed before any of the other activities may proceed, since the 
information it contains is used by the other activities to complete their tasks. It may, however, be 
5 updated at a later time by the ongoing service activities including the query response tasks. The 
search is set up utilizing the target entity qualification factors 104 to decide which data records to 
retrieve and the entity profile template 108 to select and format the data for each individual or 
entity. It reserves the fields for whatever polling, verification, transaction, or financial 
information is required. This database is the repository for all of the information generated by 
1 0 the system about the target entities. 

The next activity in preliminary services is the management of the query and response 
interface 208 with the target user 24. When present as part of the campaign, this activity allows 
two-way communication on a limited basis with the individual who receives the message. Use 
of this feature may be implemented by the media distribution network, such as a set-top box 
\f\ 15 interface, for example web TV. Preferably, the interface can distinguish the identity of the 
' p sender and associate his response with his data profile. The timing of any user interaction is 
coordinated by the master task schedule 308. 

The process of determining the message template 400 is described below. The message 
=: J template 400 embodies the rules for assembling personalized messages utilizing the profiles data 
20 and the resource library. The message template process relies on the range and detail of the 

formatted profile information, the campaign plan example message, and the requirement for any 
specified query/response activity. 

Once a message template has been constructed 400, then a resource library must be 
created 300. This library contains the alternative clips of video and audio needed to assemble the 
25 whole range of different messages. One sub-process assembles the whole range of different 

messages. Another sub-process receives environmental information 306 and synthesizes 304 the 
video and audio segments required by the message template for timely individualized message 
assembly. Another sub-process will provide the ability to synthesize 302, on the fly, artificial 
speech and visual constructs to meet the individual specifications indicated by the different entity 
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profiles. 

A separate process running in parallel obtains and maintains the environmental 
information 306 by constantly polling the appropriate information channels and making the 
acquired data available, on demand, to the synthesizer process 304 for the resource library 300. 
5 This information could be obtained fi"om places such as the Internet, the weather service, a 
private intranet, local sensors, etc. 

The information gathered in the campaign plan 100 must be formatted such that it 
outlines the process flow and coordination requirements for each step of the ongoing service 
project fi-om initiation to completion. This data will be used to establish the master task schedule 
10 308 which controls and coordinates all of the activities and events as soon as the project "clock" 
is started. 

After the preliminary service activities have been executed (except for timely updates) the 
;i3 schedule "clock" is started. This controls the scheduled generation and delivery of the 

:£ dynamically customized advertisements to all of the profile entities in the target entities profiles 

i.h 

\ 'n 1 5 and status database 204. Execution is a continuous process of "step and repeat" which can 
' y deUver messages on demand or else accumulate appropriate batches for simultaneous delivery. 

Each step begins with the presentation 206 of the next profile to the assembly node 404 
, ? which uses this profile to select and assemble the correct material fi-om the resource library 300 
!r[ into its "slot" in the message template 400. It then sends the assembled message on to the 
^ 20 transcoding and compressing operation 406 which dehvers a video stream that matches the 

specifications for the delivery media. These video messages are then transported to the chosen 
media distribution center 212 for delivery at the scheduled time to the media user interface 214 
which communicates with each appropriate target user 24. The process is then repeated for the 
next profile 206 until all Hsted profiles are serviced. 
25 Fig. 6 illustrates a sample message template 56. The message template(s) 56 design is 

determined using information fi-om the default message example 102 Fig. 5 and the general 
criteria that defines the entity profile template 108 together with any interactive information 
relevant to the message. The range of values for each media segment derives initially from the 
information given in the campaign plan 100 about the default message example 102 and its 



-20- 



in 



variations. Beyond that, the entity profile template 108 establishes the breadth of information 
sought for each profile category such that it will provide adequate information to customize the 
desired segments for the chosen entity. In addition, some information can be derived fi-om the 
interactive responses of the client entity as well as fi-om the updated environment status 
5 information provided to the message resource library databases 26. The message template 56 Fig. 
6 is the mold into which the proper selection of resources are "poured" to form the specific 
message for the specific entity whose profile has been presented for assembly. 

The message template or templates 56, together with the breadth of information 
established as previously described, forms the basis for generating the message resource library 
10 26. Each defined media segment 54 for each video and audio track of the message template 56 
requires at least one or more alternate segments 54 to be generated to provide the ftill range of 
variations required to cover all of the profile categories and the variation in category types. 
Alternatively, the message resource library can be updated with new alternate segments 54 and 
the expert rules 38 updated to provide finer granularity in customization based on feedback fi-om 



in 

in 15 inifial use of the campaign. The expert rules 38 according to the present invention allow for 



ry 



continuous tweaking of the message assembly. The message resource library also keeps updated 
information on any current environmental status information needed to assemble the messages. 

These video and audio segments will generally be stored on high capacity, high speed 
servers 200 Fig. 5, at the highest quality level needed by any of the target media through which 



O 

|;3 20 the messages will be distributed. They are available to the assembly operation in "real time" for 
production "on the fly". Each segment 54 is sized to fit its template slot and have its own 
identity and time code location for rapid retrieval by the assembly operation. In another 
embodiment, alternative segments for one template slot can vary in length, and the system 
automatically adjusts the final message to properly fit in the total time slot. Typically this is 
25 performed by shortening a following segment, which may be created to allow such variations in 
length. 

The next profile 60 of a user 24, Fig. 5 for whom the current message is being customized 
is pulled from the target entities profiles & status database 503 that contains all of the data 
collected to "fill" the profile template 60 Fig. 8 for each of the entities selected for the current 
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campaign. This profile 60 need not be limited to passive accretion of facts but may include 
information derived by sophisticated behavior analyze of buying habits etc. Each profile 60 is 
presented to the assembly operation to be used as a basis for selecting the media segments 54 that 
are appropriate to this specific profile to be placed in each of the slots in the message template 
5 56. 

The assembly operation 404 proceeds in a rudimentary fashion by selecting each media 
segment 54 fi"om its altemates by simply locating the segment in each category that includes the 
profile's value for that category and using the default segment when no information is available 
for that slot, as shown in Fig. 8. Altematively, a more powerful form of selection is used based 
10 on a logical rules to evaluate several pieces of information for any selection including how to 
deal with gaps in the profile by evaluating and interpolating other available information. Some 
example computer code written in a variation of the Lisp programming language is presented in 

C= the Appendix. 

in 

The first stage of completion results in an edit decision list which is a set of instructions 

in 

i'r^ 15 for the assembly of this specific message by a video editing fimction using the message resource 

't: r 

• z library 26. This causes a high quality video stream of the message to be generated which is then 
sent together with its ID to the transcoding operation, which is described below. 

Another embodiment of the present invention for delivering personalized messages is 
shown in Fig. 9. This embodiment is an example of personalized message creation as a service 
^ 20 industry, where clients send requests electronically to a service provider, who creates the 

personalized message and sends the personalized message either back to the client, or directly to 
the user. The personalized message may be displayed to the user immediately, or altematively, 
the personalized message may be stored by the requesting client to provide to the user at a later 
time, either by web page or by other delivery methods 155, such as hard media (video tapes, 
25 video disks, dvd etc), or by electronic means (email, cable TV, satellite etc.). 

The cHent 130 sends a request, including a user identification to the server 134. The user 
identification 132 provides information identifying the user, to allow for searching and obtaining 
user profile information. A standard user identification such as a user email address, domain 
number or social security number may be used, or a specific user identification used or created 
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by the client. The server sends a request to the system 20 which is providing the service. The 
server 1 34 also indicates a particular campaign to be used for creating the user-specific 
personalized message. A campaign scheduler 142 sets up and queues the creation of the 
personalized message by the system 20. 
5 Upon conmiencement, the profile server 503 gathers profile information 60 on the user. 

The profile server 503 can obtain profile information fi'om many sources, including databases 
200, as previously described. The profile server 503 then sends the gathered information 
including the user profile information 60, message template 56 information and specific 
campaign rule information to the message generator 145. The message generator 145 uses expert 
10 rules 38 as well as other sources of environmental and temporal data 147 (such as weather, time, 
current events etc.) to select appropriate media segments for completion of the message template 
56. The message generator then sends a media playout list 148 to the assembly system 36, which 
•iB obtains the appropriate media segments firom the databases 26 and produces the personalized 

message. Depending on the delivery requirements, the personalized message may be sent 
Jt; 1 5 directly to the delivery target 1 55, or may be encoded by a media encoder 40 which performs any 
^ y conversion or compression necessary to produce a deliverable personalized message. 

Another embodiment of the present invention is shown in Fig. 10. This embodiment is 

Q 

, r similar to the embodiment in Fig. 9, except with more focus on message creation and delivery 

over the internet or world wide web. This is an example of "just in time" creation on demand, to 

C3 20 display to a user who is actively viewing a web site on which the personalized message will be 
immediately displayed. 

Turning to Fig. 10, A client browser 130 sends a user identification 132 to the web server 
134. The user identification 132 is sent along with a campaign identification, which indicates 
what campaign to use to create the personalized message for the user. 
25 The web server 134 passes the received information on to the transaction processor 138, 

typically over an electronic connection such as the intemet, a dedicated telephone connection etc. 
The transaction processor 138 oversees much of the operation of creating the personalized 
message, including sequencing of many of the steps in the operation, and bookkeeping for client 
records and billing. For creating the personalized message, the transaction processor 138 send 
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the user identification 132 to the profile server 503, which gathers profile information 60 on the 
user, as previously described. This profile information 60 is returned to the transaction processor 
138. 

The transaction processor 138 sends the campaign identification to the campaign 
scheduler 142, v^hich schedules and sets up tasks and actions to be performed in the personalized 
message creation. The campaign scheduler controls the performance of the message generator 
145. The message generator 145 receives the profile information 60 and fiirther information 144 
(such as parameters associated with the campaign) fi-om the transaction processor 138. 

The message generator 145 provides the profile information 120 and message template 56 
to the expert rules system 38, which is responsible for completing the message template 56 with 
selected media segments for each part of the template. The expert rules system 38 uses rules as 
previously described to determine the appropriate media segments. The completed template 146 
is then retumed to the message generator 145 which sends a media play out list 148 to the 
transaction processor 138. 

The transaction processor 138 provides the media playout hst 148 to the media server 149 
which assembles all the media segments together to produce the video stream 150 of the 
personalized message. The video stream 150 is provided to the media encoder 40, to properly 
encode the video stream for delivery. In the present embodiment, this is an encoded video 
stream 154 which is sent to the chent browser 130, for delivery to the user. 

In this embodiment, the transaction processor 138, profile server 503, campaign scheduler 
142, message generator 145 expert rules server 38, media server 149 and media encoder 40 run 
on separate general purpose computers running Windows NT or Linux. These computers are 
intercormected as appropriate for the location of each component and the bandwidth 
requirements for communication therebetween. Alternatively, several components may run on 
one machine as one application, or as separate processes. 

In an alternative embodiment, the present invention may be implemented with a plurality 
of message templates for one campaign, with a template selection mechanism for selecting an 
appropriate template based on profile information for the end user, or on other factors. For 
example, if it is presently raining in a location where the end user is, a message template created 
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with bad weather in mind may be selected over a "good" weather message template, or a generic 
template. These plurality of message templates may have common insertable message segments, 
or use different libraries of insertable message segments, or even have few or no insertable 
message segments. 

Although the invention has been shown and described with respect to illustrative 
embodiments thereof, various other changes, omissions and additions in the form and detail 
thereof may be made therein without departing from the spirit and scope of the invention. 
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